Image processing device, image processing program, and program-recording medium

ABSTRACT

The present invention provides an image processing apparatus, comprising: an image signal inputting unit  11  for inputting a plurality of image signals different from one another in exposure time, first and second level converting factor calculating units  12, 13  for calculating a level converting factor for each of the image signals, a weighting factor calculating unit  14  for calculating a weighting factor for each of the image signals; and an adding unit  17  for adding up the image signals respectively multiplied by the level converting factors multiplied by the weighting factors. The image processing apparatus thus constructed can obtain images of wide dynamic range in a simply manner as well as reduce loss of minute image information due to accumulated processing delays and computing errors.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to an image processing apparatus forprocessing a digital image signal, and more particularly to an imageprocessing apparatus for and an image processing program for, and astorage medium having stored therein the image processing program forobtaining images of a wide dynamic range.

DESCRIPTION OF THE RELATED ART

Up until now, there have been provided a wide variety of prior-art imageprocessing apparatuses. One typical example of the conventional imageprocessing apparatus of this type is disclosed in, for example, JapanesePatent Laid-Open Publication No. H06-141229.

The conventional image processing apparatus is exemplified and shown inFIG. 9. As best shown in FIG. 9, the conventional image processingapparatus comprises an imaging device 1, a memory unit 2, a multiplyingunit 3, a first level weighting unit 4, a second level weighting unit 5,an adding unit 6, a rate converting unit 7, a compressing unit 8, and atiming control unit 9. The imaging device 1 is adapted to take aplurality of images of a specific object for short and long exposuretimes to transform the images into image signals. The image signalindicative of an image taken by the imaging device 1 for the shortexposure time will be hereinlater referred to simply as “image signal ofthe short exposure time”. The image signal indicative of an image takenby the imaging device 1 for the long exposure time will be hereinlaterreferred to simply as “image signal of the long exposure time”. Theimaging device 1 is operative to output each of the image signals of theshort and long exposure times alternately therethrough for every onefield. The image signals of the short and long exposure times areprocessed and synthesized before being outputted therethrough. Thememory unit 2 is operative to store therein the image signal of theshort exposure time. The timing control unit 9 is operative to controlthe memory unit 2 to have the memory unit 2 output therethrough theimage signal of the short exposure time at a timing synchronized withthe image signal of the long exposure time outputted from the imagingdevice 1. The multiplying unit 3 is adapted to multiply the image signalof the short exposure time by the ratio of the long exposure time to theshort exposure time to output the image signal of the short exposuretime thus multiplied to the second level weighting unit 5. The secondlevel weighting unit 5 is operative to multiply the image signal of theshort exposure time inputted from the multiplying unit 3 by a weightingfactor calculated in accordance with the signal level of the imagesignal of the short exposure time. Meanwhile, the first level weightingunit 4 is operative to multiply the image signal of the long exposuretime inputted from the imaging device 1 by a weighting factor calculatedin accordance with the signal level of the image signal of the longexposure time. The adding unit 6 is operative to add up the image signalof the short exposure time weighted by the second level weighting unit 5and the image signal of the long exposure time weighted by the firstlevel weighting unit 4. The image signals thus added are outputted fromthe adding unit 6 at a high rate faster than a usual rate because of thefact that the image signals have been read out at the imaging device 1at the high rate so as to be later processed as image signals of theshort and long exposure times. The rate converting unit 7 is operativeto input the image signals at an input rate from the adding unit 6 tooutput the image signals at an output rate appropriate to a processingrate of an output unit such as, for example, a monitor. The compressingunit 8 is operative to compress each of the image signals inputted fromthe rate converting unit 7 in a signal level in accordance with adynamic range of the output unit in such a manner that the image signalshaving signal levels higher than a predetermined level are compressed tothe image signals having signal levels below the predetermined level tobe outputted to the output unit such as, for example, a monitor.

The conventional image processing apparatus thus constructed aspreviously mentioned can provide the images of a wide dynamic range onthe monitor so as to prevent dark portions of the object displayed on,for example, the monitor from becoming monotonously black and ambiguousand bright portions of the object displayed on the monitor from becomingmonotonously white resulting from the fact that the image signals havinghigh signal levels have been nonlinearly compressed by the compressingunit 8 to the image signals of signal levels below a predeterminedlevel, and the monitor is operative to display the dark portions of theobject on the basis of the image signals of the long exposure time andthe bright portions of the object on the basis of the image signals ofthe short exposure time.

The conventional image processing apparatus thus constructed aspreviously mentioned, however, encounters a drawback in that complicatesprocesses are required, the number of bits forming the image signal isincreased, and thus circuits forming part thereof are increased in sizeresulting from the fact that the image signal of the short exposure timeis multiplied by the ratio of the long exposure time to the shortexposure time so that the signal level of the image signal of the shortexposure time become close to that of the image signal of the longexposure time.

Further, the conventional image processing apparatus thus constructed aspreviously mentioned encounters another drawback in that processingdelays are increased, and minute image information tends to be lost dueto accumulated computing errors because of the fact that many computingoperations are required to be carried out on the image signals.

Still further, the conventional image processing apparatus thusconstructed as previously mentioned encounters another drawback in thatthe circuits forming part thereof are further increased in size in theevent that the color image signals are processed by the reason that eachof the color image signals is divided into a brightness signal and acolor-difference signal and the color-difference signals are controlledon the basis of rate of nonlinear change of the brightness signals andtherefore the rate of change of the brightness signals is required to becalculated.

The present invention is made for the purpose of overcoming the abovementioned drawbacks, and it is therefore an object of the presentinvention to provide an image processing apparatus, an image processingprogram, and a storage medium having stored therein the image processingprogram for obtaining image of a wide dynamic range in a simple mannerwhile minimizing the processing delays and loss of minute imageinformation due to accumulated computing errors.

DISCLOSURE OF THE INVENTION

In accordance with a first aspect of the present invention, there isprovided an image processing apparatus, comprising: image signalinputting means for inputting a plurality of image signals differentfrom one another in exposure time; level converting factor calculatingmeans for calculating a level converting factor for each of the imagesignals; weighting factor calculating means for calculating a weightingfactor for each of the image signals; and adding means for adding up theimage signals respectively multiplied by the level converting factorsmultiplied by the weighting factors.

The image processing apparatus according to the present invention thusconstructed can provide high quality images of a wide dynamic range to,for example, a monitor so as to prevent dark portions of the objectdisplayed on the monitor from becoming monotonously black and ambiguousand bright portions of the object displayed on the monitor from becomingmonotonously white in a simple manner without increasing the size ofcircuits forming part thereof as well as reduce loss of minute imageinformation due to accumulated processing delays and computing errors.

Further, in the image processing apparatus according to the presentinvention, the level converting factor calculating means may beoperative to calculate the level converting factor for each of the imagesignals in such a manner that the image signals respectively multipliedby the level converting factors are substantially equal to one anotherin signal level under the condition that the signal levels of the imagesignals are less than a predetermined value, and the image signalsrespectively multiplied by the level converting factors are reduced insignal level under the condition that the signal levels of the imagesignals are above the predetermined value.

The image processing apparatus according to the present invention thusconstructed can reduce the number of computations carried out on theimage signals, thereby making it possible for the image processingapparatus to be in a simple construction without increasing the size ofthe circuits forming part thereof.

Further, the image processing apparatus may further comprise: highfrequency component extracting means for allowing only high frequencycomponents of the image signals greater than a predetermined frequencylevel to be passed therethrough; and high frequency level convertingfactor calculating means for calculating a high frequency levelconverting factor for each of the high frequency components of the imagesignals passed by the high frequency component extracting means, and inwhich the adding means may be operative to multiply by the weightingfactor the sum of each of the image signals multiplied by the levelconverting factor and the high frequency component of each of the imagesignals multiplied by the high frequency level factor.

The image processing apparatus according to the present invention thusconstructed can provide high quality images of a wide dynamic range frombright portions to dark portions with sharpness using the high frequencycomponents of the image signals in a simple manner without increasingthe size of circuits forming part thereof.

Further, in the image processing apparatus, each of the image signalsmay include color components, and which may further comprise:representative value calculating means for calculating a representativevalue for the color components for each of the image signals, and inwhich the level converting factor calculating means may be operative tocalculate a level converting factor for each of the image signals on thebasis of the representative value, and the weighting factor calculatingmeans may be operative to calculate a weighting factor for each of theimage signals on the basis of the representative value.

The image processing apparatus according to the present invention thusconstructed can employ the same level converting factor for each ofrepresentative values and thus obtain high quality color images of awide dynamic range from bright portions to dark portions with the hue ofthe color images maintained constant as well as prevent circuits formingpart thereof from being increased in size.

Further, in the image processing apparatus, the image signal inputtingmeans may be operative to input a plurality of image signals from animaging device operative to temporarily store therein image signals ofrespective exposure times for respective time periods in such a mannerthat the image signals are synchronized with one another.

The image processing apparatus according to the present invention thusconstructed can provide high quality images of a wide dynamic range to,for example, a monitor so as to prevent dark portions of the objectdisplayed on the monitor from becoming monotonously black and ambiguousand bright portions of the object displayed on the monitor from becomingmonotonously white in a simple manner.

In accordance with a second aspect of the present invention, there isprovided a program executable by an image processing apparatus to carryout: an image signal inputting step of inputting a plurality of imagesignals different from one another in exposure time; a level factorcalculating step of calculating a level converting factor for each ofthe image signals; a weighting factor calculating step of calculating aweighting factor for each of the image signals; and an adding step ofadding up the image signals respectively multiplied by the levelconverting factors multiplied by the weighting factors.

The program according to the present invention thus constructed makes itpossible for a microprocessor, a digital signal processor, or the liketo carry out image processing steps to obtain high quality images of awide dynamic range.

Further, in the program according to the present invention, the levelfactor calculating step is of calculating the level converting factorfor each of the image signals in such a manner that the image signalsrespectively multiplied by the level converting factors aresubstantially equal to one another in signal level under the conditionthat the signal levels of the image signals are less than apredetermined value, and the image signals respectively multiplied bythe level converting factors are reduced in signal level under thecondition that the signal levels of the image signals are above thepredetermined value.

The program according to the present invention thus constructed canreduce the number of computations carried out on the image signals, andthus makes it possible for a machine executing the program to obtainhigh quality images of a wide dynamic range in a simple manner whilepreventing the size of circuits forming part of the machine from beingincreased.

Further, in the program according to the present invention, the levelfactor calculating step may be of calculating the level convertingfactor for each of the image signals on the basis of predetermined datarepresentative of level converting factor characteristics.

The program according to the present invention thus constructed makes itpossible for a machine executing the program to obtain level appropriateconverting factors in a simple manner.

In accordance with a third aspect of the present invention, there isprovided a storage medium having stored therein a program executable byan image processing apparatus according to the present invention tocarry out: an image signal inputting step of inputting a plurality ofimage signals different from one another in exposure time; a levelfactor calculating step of calculating a level converting factor foreach of the image signals; a weighting factor calculating step ofcalculating a weighting factor for each of the image signals; and anadding step of adding up the image signals respectively multiplied bythe level converting factors multiplied by the weighting factors.

The storage medium having stored therein a program according to thepresent invention thus constructed makes it possible for amicroprocessor, a digital signal processor, or the like to carry outimage processing steps to obtain high quality images of a wide dynamicrange.

Further, in the storage medium having stored therein a programexecutable by an image processing apparatus according to the presentinvention, the level factor calculating step may be of calculating thelevel converting factor for each of the image signals in such a mannerthat the image signals respectively multiplied by the level convertingfactors are substantially equal to one another in signal level under thecondition that the signal levels of the image signals are less than apredetermined value, and the image signals respectively multiplied bythe level converting factors are reduced in signal level under thecondition that the signal levels of the image signals are above thepredetermined value.

The storage medium having stored therein a program according to thepresent invention thus constructed can reduce the number of computationscarried out on the image signals, and thus makes it possible for amachine executing the program to make the image signals substantiallyequal to one another in signal level under the condition that the signallevels of the image signals are less than a predetermined value, and theimage signals reduced in signal level under the condition that thesignal levels of the image signals are above the predetermined value.

Further, in the storage medium having stored therein a programexecutable by an image processing apparatus according to the presentinvention, the level factor calculating step may be of calculating thelevel converting factor for each of the image signals on the basis ofpredetermined data representative of level converting factorcharacteristics.

The storage medium having stored therein a program according to thepresent invention thus constructed makes it possible for a machineexecuting the program to obtain level appropriate converting factors ina simple manner.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of an image processing apparatus and animage processing program according to the present invention will be moreclearly understood from the following description taken in conjunctionwith the accompanying drawings in which:

FIG. 1 is a block diagram showing a first preferred embodiment of theimage processing apparatus according to the present invention;

FIG. 2 is a graph showing signal levels in relation to brightness levelof an object;

FIG. 3(a) is a graph showing a level compressing function applied toimage signals indicative of images taken for long exposure time;

FIG. 3(b) is a graph showing level converting factors in relation toimage signals indicative of images taken for long exposure time;

FIG. 3(c) is a graph showing a level compressing function applied toimage signals indicative of images taken for short exposure time;

FIG. 3(d) is graph showing level converting factors in relation to imagesignals indicative of images taken for short exposure time;

FIG. 4 is a graph showing signal levels of image signals multiplied bythe level converting factors respectively shown in FIGS. 3(c) and 3(d);

FIG. 5 is graph showing weighting factors calculated by a weightingfactor calculating unit forming part of the image processing apparatusshown in FIG. 1;

FIG. 6 is a block diagram showing a second preferred embodiment of theimage processing apparatus according to the present invention;

FIG. 7 is a block diagram showing a third preferred embodiment of theimage processing apparatus according to the present invention;

FIG. 8 is a block diagram showing an imaging apparatus applicable to theimage processing apparatus according to the present invention; and

FIG. 9 is a block diagram showing a conventional image processingapparatus.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

A preferred embodiment of the present invention will be describedhereinafter with reference to the drawings.

FIG. 1 is a block diagram showing a first preferred embodiment of animage processing apparatus according to the present invention.

As will be clearly seen from FIG. 1, the first embodiment of the imageprocessing apparatus according to the present invention comprises imagesignal inputting means constituted by an image signal inputting unit 11for inputting a plurality of image signals including image signal eachtransformed from images of a specific object taken for a short exposuretime, hereinlater referred to simply as a “short image signal” and animage signal transformed from an image of a specific object taken for along exposure time, hereinlater referred to simply as a “long imagesignal”, a first level converting factor calculating unit 12 forcalculating a level converting factor for each of the long image signalsin such a manner that the long image signals respectively multiplied bythe level converting factors are substantially equal to one another insignal level under the condition that the signal levels of the longimage signals are less than a predetermined value, and the long imagesignals respectively multiplied by the level converting factors arereduced in signal level under the condition that the signal levels ofthe long image signals are above the predetermined value, and a secondlevel converting factor calculating unit 13 for calculating a levelconverting factor for each of the short image signals in such a mannerthat the short image signals respectively multiplied by the levelconverting factors are substantially equal to one another in signallevel under the condition that the signal levels of the short imagesignals are less than a predetermined value, and the short image signalsrespectively multiplied by the level converting factors are reduced insignal level under the condition that the signal levels of the shortimage signals are above the predetermined value. The first levelconverting factor calculating unit 12 and the second level convertingfactor calculating unit 13 collectively constitute level convertingfactor calculating means.

The image processing apparatus further comprises weighting factorcalculating means constituted by a weighting factor calculating unit 14for calculating a weighting factor for each of the long image signalsand the short image signals, a first multiplying unit 15 a formultiplying the level converting factor calculated by the first levelconverting factor calculating unit 12 by the weighting factor calculatedby the weighting factor calculating unit 14, and a second multiplyingunit 15 b for multiplying the level converting factor calculated by thesecond level converting factor calculating unit 13 by the weightingfactor calculated by the weighting factor calculating unit 14.

The image processing apparatus further comprises a third multiplyingunit 16 a for multiplying each of the long image signals by the productof the level converting factor calculated by the first level convertingfactor calculating unit 12 and the weighting factor calculated by theweighting factor calculating unit 14 calculated by the first multiplyingunit 15 a, a fourth multiplying unit 16 b for multiplying each of theshort image signals by the product of the level converting factorcalculated by the second level converting factor calculating unit 13 andthe weighting factor calculated by the weighting factor calculating unit14 calculated by the second multiplying unit 15 b, and adding meansconstituted by an adding unit 17 for adding up each of the long imagesignals multiplied by the third multiplying unit 16 a and the shortimage signals multiplied by the fourth multiplying unit 16 b.

The operation of the first embodiment of the image processing apparatusaccording to the present invention will be described hereinlater.

The image signal inputting unit 11 is operated to input a plurality ofimage signals different from one another in exposure time. The imagesignals inputted by the image signal inputting unit 11 are synchronizedwith one another so to be processed simultaneously, and include longimage signals and short image signals. Each of the image signals has asignal level. The first level converting factor calculating unit 12 isoperated to calculate a level converting factor for each of the longimage signals, and the second level converting factor calculating unit13 is operated to calculate a level converting factor for each of theshort image signals. The operations performed by the first levelconverting factor calculating unit 12 and the second level convertingfactor calculating unit 13 will be described later.

The weighting factor calculating unit 14 is operated to calculate aweighting factor for each of the long image signal and the short imagesignal on the basis of the signal level of each of the long imagesignal. The operation performed by the weighting factor calculating unit14 will be described later.

The first multiplying unit 15 a is operated to multiply the levelconverting factor calculated by the first level converting factorcalculating unit 12 by the weighting factor calculated by the weightingfactor calculating unit 14. The third multiplying unit 16 a is thenoperated to multiply each of the long image signals by the product ofthe level converting factor calculated by the first level convertingfactor calculating unit 12 and the weighting factor calculated by theweighting factor calculating unit 14 calculated by the first multiplyingunit 15 a with the result that the long image signals thus calculatedare substantially equal to one another in signal level under thecondition that the signal levels of the long image signals are less thana predetermined value, and the long image signals thus calculated arereduced in signal level under the condition that the signal levels ofthe long image signals are above the predetermined value, as well asweighted by the weighting factor. Likewise, the second multiplying unit15 b is operated to multiply the level converting factor calculated bythe second level converting factor calculating unit 13 by the weightingfactor calculated by the weighting factor calculating unit 14. Thefourth multiplying unit 16 b is then operated to multiply each of theshort image signals by the product of the level converting factorcalculated by the second level converting factor calculating unit 13 andthe weighting factor calculated by the weighting factor calculating unit14 calculated by the second multiplying unit 15 b, with the result thatthe short image signals thus calculated are substantially equal to oneanother in signal level under the condition that the signal levels ofthe short image signals are less than a predetermined value, and theshort image signals thus calculated are reduced in signal level underthe condition that the signal levels of the short image signals areabove the predetermined value, as well as weighted by the weightingfactor.

The adding unit 17 is operated to add up each of the long image signalsmultiplied by the third multiplying unit 16 a and the short imagesignals multiplied by the fourth multiplying unit 16 b and output theimage signals thus added up to, for example, a monitor.

The following description will be directed to a method of calculatingthe level converting factors for the long image signal and the shortimage signal.

FIG. 2 is a graph showing signal levels in relation to brightness levelsof the object, wherein the lateral axis indicates the brightness levelof the object and the longitudinal axis indicates the signal level ofthe image signal.

As will be clearly seen from FIG. 2, the signal level of the long imagesignal denoted by a reference numeral 21 is saturated substantially to asaturated level 23 under the condition that the brightness level of theobject is increased above a predetermined upper brightness level, andthe gradient of the signal level of short image signal denoted by areference numeral 22 is smaller than the gradient of the signal level ofthe long image signal 21 in correlation with a ratio of respectiveexposure times. This means that an image displayed by, for example, themonitor based on the long image signals 21 indicative of portions of theobject whose brightness level is above the upper brightness levelbecomes monotonously white and an image display by the monitor based onthe short image signals 22 indicative of portions of the object whosebrightness level is low becomes monotonously dark and ambiguous.

The present embodiment of the image processing apparatus according tothe present invention is operative to synthesize the long image signals21 and the short image signals 22 in accordance with the brightnesslevels of the object to obtain high quality images of a wide dynamicrange so as to be displayed on, for example, a monitor as an image ofthe object having bright and dark portions to be clearly seen by aviewer. The long image signals 21 and the short image signals 22 arerequired to be processed before being synthesized, for the purpose ofappropriately adjusting the signal levels of the long image signals 21and the short image signals 22. In the case that each of the short imagesignals 22 is just multiplied by a ratio of the long exposure time tothe short exposure time, the signal levels of the short image signals 22thus multiplied, however, may increase above a predetermined uppersignal level and could not be displayed by the monitor because of thefact that the signal levels of the image signals above the upper signallevel would require a large number of bits while the output unit suchas, for example, a monitor could display image signals only with thelimited number of bits. Accordingly, the signal levels of the imagesignals thus multiplied should be reduced in the number of bits to anappropriate level.

This leads to the fact that, in the present embodiment, the signal levelof each of the image signals above the predetermined upper signal levelare nonlinearly compressed to have the number of the bits forming partof each of the image signals compressed to an appropriate number.

FIGS. 3(a) to (d) are graphs each showing characteristic data of levelcompressing function and level converting factors applied to the longimage signals and the short image signals.

FIG. 3(a) is a graph showing characteristics of a level compressingfunction applied to the long image signals 21.

The level compressing function denoted by a reference numeral 31 isdetermined by performing the computation represented by an expression 1as follows.Y=Flong(X)  (Expression 1)

X: inputted image signals

Wherein “F long (X)” is intended to mean the level compressing function31, and “Y” is intended to mean the value of the computation.

The long image signals are computed in accordance with the levelcompressing function 31. This means that the level compressing function31 can be arbitrary specified so as to make the long image signals havecharacteristics appropriate for the output unit such as a monitor. Thelevel compressing function 31 may be specified so that, for example, thesignal levels of the long image signals thus computed in accordance withthe level compressing function 31 are drastically increased under thecondition that the signal levels of the long image signals are low, andthe signal levels of the long image signals are reduced below thepredetermined saturated level under the condition that the signalslevels of the long image signals are above the predetermined saturatedlevel. The image of the object displayed by, for example, the monitorbased on the image signals thus computed can be clearly seen by a viewerbecause of the fact that the dark portions of the object are displayedon the basis of the image signals having signal levels thus increasedand the bright portions of the object are displayed on the basis of theimage signals thus compressed below the predetermined saturated level.

Further, in view of an object and its intended use, the specified levelcompressing function 31 may be in the form of, for example, an S shape.The image signals thus computed can be advantageous especially when theimage of the object displayed on the basis of the long image signalshaving middle signal levels is required to be clearly seen. From thedescription in the above, it is understood that the characteristics ofthe level compressing function 31 can be arbitrary specified utilizing,for example, a microcomputer in view of an object and its intended useto obtain desired image signals of high quality.

FIG. 3(b) is a graph showing characteristics of level converting factorsin relation to the long image signals 21.

The level converting factors denoted by a reference numeral 32 aredetermined in combination with the level compressing function 31 byperforming the computation represented by an expression 2 as follows.Along(X)=Flong(X)/X  (Expression 2)

Wherein “A long (X)” is intended to mean the level converting factors 32and “F long (X)” is intended to mean the level compressing function 31.

It will be understood that the long image signals multiplied by thelevel converting factors 32 results in the values equal to those of thelong image signals computed in accordance with the level compressingfunction 31 as will be clearly seen from an expression 3 describedbelow.Along X)*X=Flong (X)/X*X=Y  (Expression 3)

This leads to the fact that high signal levels of the long image signalscan be compressed after the long image signals are multiplied by thelevel converting factors 32.

FIG. 3(c) is a graph showing characteristics of a level compressingfunction applied to short image signals 22.

Similar to the description in the above about the long image signals,the level compressing function denoted by a reference numeral 33 isdetermined by performing the computation represented by an expression 4as follows.Y=Fshort(X)  (Expression 4)

X: inputted image signals

Wherein “F short (X)” is intended to mean the level compressing function33, and “Y” is intended to mean the value of the computation.

Here, the signal levels of the short image signals can be adjusted insuch a manner that the signal levels of the short image signals becomeclose to those of the long image signals by performing the computationrepresented by an expression 5 as follows.

FIG. 3(d) is a graph showing characteristics of level converting factorsin relation to the short image signals 22.

Similar to the description in the above about the long image signals,the level converting factors denoted by a reference numeral 34 aredetermined in combination with the level compressing function 33 byperforming the computation represented by an expression 6 as follows.Ashort(X)=Fshort(X)/X  (Expression 6)

Wherein “A short (X)” is intended to mean the level converting factors34 and “F short (X)” is intended to mean the level compressing function33.

It will be understood that the short image signals multiplied by thelevel converting factors 34 results in the values equal to those of theshort image signals computed in accordance with the level compressingfunction 33 as will be clearly seen from an expression 7 describedbelow.Ashort(X)*X=Fshort(X)/X*X=Y  (Expression 7)

This leads to the fact that the signal levels of the short image signalscan be adjusted in such a manner that the signal levels of the shortimage signals become close to those of the long image signals as well asthe signal levels of the short image signals are compressed bymultiplying the short image signals by the level converting factors 34.Here, it is not necessary to make the signal levels of the short imagesignals become exactly equal to those of the long image signals.

The level converting factors 32 and 34 respectively shown in FIGS. 3(b)and 3(d) may have been calculated in advance by, for example, amicrocomputer, and stored in, for example, a memory as being in theformat of a table, and the present embodiment of the image processingapparatus may obtain the level converting factors 32 and 34corresponding to the signal levels of the long image signal and theshort image signal with reference to the table stored in the memory.

It is more preferable in view of saving a memory space that a pluralityof points representative of the level converting factors 32 and 34respectively shown in FIGS. 3(b) and 3(d) may have been selected inadvance and stored in, for example, a memory as being in the format of atable, and the present embodiment of the image processing apparatus mayobtain the level converting factors 32 and 34 corresponding to thesignal levels of the long image signal and the short image signal withreference to the table stored in the memory, if necessary, interpolatingthe points.

FIG. 4 is a graph showing characteristics of signal levels of imagesignals multiplied by the level converting factors respectively shown inFIGS. 3(c) and 3(d) wherein the lateral axis indicates the inputtedimage signal before the level converting process and the longitudinalaxis indicates the outputted image signal after the level convertingprocess. As will be clearly seen from FIG. 4, it is to be understoodthat the signal levels of the outputted image signals, viz., both theoutputted long image signal 41 and the outputted short image signal 42are drastically increased under the condition that the signal levels ofthe inputted image signals are low, and the signal levels of theoutputted image signals are reduced below the predetermined saturatedlevel under the condition that the signals levels of the inputted imagesignals are above the predetermined saturated level. Further, it is tobe understood that the long image signal 41 and the short image signal42 are substantially identical in signal level under the condition thatthe signal levels of the inputted image signals are low.

The following description will be directed to a method of calculatingthe weighting factors for the long image signal and the short imagesignal.

FIG. 5 is a graph showing weighting factors calculated by the weightingfactor calculating unit 14.

The weighting factor calculating unit 14 is operative to calculate aweighting factor 51 for the long image signal based on the signal levelof the long image signal and a weighting factor 52 for the short imagesignal based on the signal level of the short image signal in such amanner that the weighting factor 51 for the long image signal is largeunder the condition that the signal level of the inputted image signalis low and the weighting factor 52 for the short image signal is largeunder the condition that the signal level of the inputted image signalis high as will be clearly seen from FIG. 5. This makes it possible forthe present embodiment of the image processing apparatus to use the longimage signals, which are less degraded due to noises, to display thedark portions of the image and the short image signals, which are notsaturated, to display the bright portions of the image so as to obtainhigh quality image signals.

Further, the weighting factor calculating unit 14 is operative tocalculate a weighting factor 51 for the long image signal and aweighting factor 52 for the short image signal in such a manner that thesum of the weighting factor 51 for the long image signal and theweighting factor 52 for the short image signal is constantly maintainedat one while adaptively changing the weighting factor 51 for the longimage signal and the weighting factor 52 for the short image signal inresponse to their signal levels. This makes it possible for the presentembodiment of the image processing apparatus to prevent the sum of thelong image signals multiplied by the weighting factors and the shortimage signals multiplied by the weighting factors from being severelyfluctuated in signal levels and thus smoothly synthesize the long imagesignals and the short image signals based on the ratio of their signallevels.

Further, the weighting factor calculating unit 14 is designed tocalculate a weighting factor 51 for the long image signal and theweighting factor 52 for the short image signal under the condition thatthe signal level of image signal is above a predetermined crossing startlevel denoted by a reference numeral 53 with a gradient of the shortimage signals denoted by a reference numeral 54 as clearly seen fromFIG. 5. According to the present invention, the present embodiment ofthe image processing apparatus may adaptively change the crossing startlevel 53 and the gradient 54 so as to process image signals of variesconditions.

While it has been described in the above about the fact that the presentembodiment of the image processing apparatus is operative to process thelong image signals and the short image signals separately from eachother, the image processing apparatus according to the present inventionis not limited to processing the long image signals and short imagesignals separately from each other. The image processing apparatusaccording to the present invention may comprise, for example, one levelconverting factor calculating unit in place of the first levelconverting factor calculating unit 12 and the second level convertingfactor calculating unit 13 because of the fact that the long imagesignals and the short image signals are processed in the same mannerexcept for the values of the level converting factors. This means thatthe image processing apparatus may process the long image signals andthe short image signals as time division signals at a double rate, andcomprise one single level converting factor calculating unit forselectively calculating level converting factors for the long imagesignals and the short image signals switched one after another, and theweighting factor calculating unit 14 is operative to input the longimage signals only, and the adding unit 17 is operative to add up eachof the long image signals and the short image signals as well as toconvert the rate. The image processing apparatus thus constructed ispreferable in view of the size of the circuit.

Further, though it has been described herein about the fact that thepresent embodiment of the image processing apparatus is operative toprocess two types of image signals different from each other in exposuretime, viz., the long image signals and the short image signals, theimage processing apparatus according to the present invention is notlimited to processing two types of image signals. It is needless tomention that the same effect can still be obtained even when the imageprocessing apparatus according to the present invention processes, forexample, more than two types of image signals different from one anotherin exposure time.

As will be seen from the foregoing description, it will be understoodthat the present embodiment of the image processing apparatus accordingto the present invention can reduce the number of computations and stillprovide high quality images of a wide dynamic range to, for example, amonitor so as to prevent dark portions of the object displayed on themonitor from becoming monotonously black and ambiguous and brightportions of the object displayed on the monitor from becomingmonotonously white in a simple manner without increasing the size ofcircuits forming part thereof resulting from the facts that the imagesignals are multiplied by the level converting factors to besubstantially close to one another in signal level below a predeterminedsignal level and the image signals are multiplied by the weightingfactors. Further, the present embodiment of the image processingapparatus can reduce the number of computations and loss of minute imageinformation due to accumulated processing delays and computing errorsresulting from the fact that the level converting factors are multipliedby the weighting factors before the image signals are multiplied by thelevel converting factors.

Further the more, the present embodiment of the image processingapparatus can prevent circuits forming part thereof from being increasedin size due to the increase in the number of bits resulting from thefact that the image signals are multiplied by the level convertingfactors to be substantially close to one another in signal level below apredetermined signal level.

FIG. 6 is a block diagram showing a second preferred embodiment of animage processing apparatus according to the present invention.

As will be clearly seen from FIG. 6, the second embodiment of the imageprocessing apparatus according to the present invention is substantiallythe same in construction as the first embodiment of the image processingapparatus except for the fact that the second embodiment of the imageprocessing apparatus further comprises high frequency componentextracting means constituted by high pass filters, hereinlater simplyreferred to as a HPF 65 a and a HPF 65 b for allowing only highfrequency components of the image signals greater than a predeterminedfrequency level to be passed therethrough and high frequency levelconverting factor calculating means constituted by a first highfrequency level converting factor calculating unit 66 for calculating ahigh frequency level converting factor for each of the high frequencycomponents of the short image signals passed by the HPF 65 a and asecond high frequency level converting factor calculating unit 67 forcalculating a high frequency level converting factor for each of thehigh frequency components of the long image signals passed by the HPF 65b. The parts same as the first embodiment of the sound retrievalapparatus are not described in detail to avoid tedious repetition.

The following description will be directed to constituent elementsforming part of the present embodiment of the image processing apparatusdifferent from the first embodiment.

The HPF 65 a is adapted to allow only high frequency components of thelong image signals greater than a predetermined frequency level to bepassed therethrough. The HPF 65 b is adapted to allow only highfrequency components of the short image signals greater than apredetermined frequency level to be passed therethrough. The first highfrequency level converting factor calculating unit 66 is adapted tocalculate a high frequency level converting factor for each of the highfrequency components of the short image signals passed by the HPF 65 b.Here, the high frequency level converting factors may have beencalculated in advance and stored in, for example, a memory as being inthe format of a table. Further, the first high frequency levelconverting factor calculating unit 66 may calculate a high frequencylevel converting factor for each of the high frequency components of theshort image signals in such a manner that the high frequency componentsof the short image signals have appropriate characteristics differentfrom those of the short image signals multiplied by the level convertingfactors, e.g., the high frequency components of the short images havinghigh signal levels are not so much compressed. The image processingapparatus thus constructed makes it possible for the high frequencycomponents of the short image signals to remain after all of the imagesignals are added up, and thus can obtain sharp images.

Likewise, the second high frequency level converting factor calculatingunit 67 is adapted to calculate a high frequency level converting factorfor each of the high frequency components of the long image signalspassed by the HPF 65 a. The second high frequency level convertingfactor calculating unit 67 may calculate a high frequency levelconverting factor for each of the high frequency components of the longimage signals in such a manner that the high frequency components of thelong image signals have appropriate characteristics different from thoseof the short image signals multiplied by the level converting factors,e.g., the high frequency components of the long images having highsignal levels are not so much compressed. The image processing apparatusthus constructed makes it possible for the high frequency components ofthe long image signals to remain after all of the image signals areadded up, and thus can obtain sharp images.

The multiplying unit 68 a is operative to multiply the long imagesignals inputted by the image signal inputting unit 61 by the levelconverting factor calculated by the first level converting factorcalculating unit 62. The multiplying unit 69 a is operative to multiplythe high frequency components of the long image signals passed by theHPF 65 a by the high frequency level converting factors calculated bythe second high frequency level converting factor calculating unit 67.The adding unit 70 a is operative to add up the long image signalsmultiplied by the multiplying unit 68 a and the high frequencycomponents of the long image signals multiplied by the multiplying unit69 a.

Likewise, the multiplying unit 68 b is operative to multiply the shortimage signals inputted by the image signal inputting unit 61 by thelevel converting factor calculated by the second level converting factorcalculating unit 63. The multiplying unit 69 b is operative to multiplythe high frequency components of the short image signals passed by theHPF 65 b by the high frequency level converting factors calculated bythe first high frequency level converting factor calculating unit 66.The adding unit 70 b is operative to add up the short image signalsmultiplied by the multiplying unit 68 a and the high frequencycomponents of the short image signals multiplied by the multiplying unit69 b.

The multiplying unit 71 a is operative to multiply the long imagesignals added by the adding unit 70 a by the weighting factorscalculated by the weighting factor calculating unit 64. The multiplyingunit 71 b is operative to multiply the short image signal added by theadding unit 70 b by the weighting factors calculated by the weightingfactor calculating unit 64. The adding unit 72 is operative to add upthe long image signals added by the adding unit 70 a and the short imagesignals added by the adding unit 70 b and output the image signal thusadded up to, for example, a monitor for displaying the image signals.

As will be seen from the foregoing description, it will be understoodthat the present embodiment of the image processing apparatus accordingto the present invention makes it possible for the high frequencycomponents of the short image signals to remain after all of the imagesignals are added up, and thus can obtain sharp images in high frequencycomponents, viz., high quality images of a wide dynamic range as well asprevent circuits forming part thereof from being increased in size dueto the increase in the number of bits to make the image signalssubstantially close to one another as in the case of the conventionalimage processing apparatus, resulting from the fact that the imageprocessing apparatus is operative to extract high frequency componentsfrom the image signals and multiply the high frequency components byhigh frequency level converting factors appropriate for the highfrequency components.

FIG. 7 is a block diagram showing a third preferred embodiment of animage processing apparatus according to the present invention.

The third embodiment of the image processing apparatus according to thepresent invention is operative to process color image signals and, aswill be clearly seen from FIG. 7, is substantially the same inconstruction as the first embodiment of the image processing apparatusexcept for the fact that the second embodiment of the image processingapparatus further comprises representative value calculating meansconstituted by a first representative value calculating unit 82 a forcalculating a representative value for each of the long image signals,and a second representative value calculating unit 82 b for calculatinga representative value for each of the short image signals.

The following description will be directed to constituent elementsforming part of the present embodiment of the image processing apparatusdifferent from the first embodiment.

In order to process a color image, image data indicative of the colorimage, in general, is required to be inputted as, for example, primarycolor image signals indicative of red (R), green (G), and blue (B),complementary color image signals indicative of cyan (Cy), magenta (Mg),yellow (Ye), and green (G), or the like. The aforementioned color imagesignals indicative of respective color component information aredifferent from one another in signal level. This leads to the fact thatthe ratio of respective color components may be distorted when the imagesignals are nonlinearly compressed, and thus the hue of the color imagesmay be changed in the case that the first embodiment of the imageprocessing apparatus simply processes the color image signals.

In the present embodiment of the image processing apparatus according tothe present invention, the first representative value calculating unit82 a is operative to calculate a representative value for the colorcomponents for each of the long image signals, and the secondrepresentative value calculating unit 82 b is operative to calculate arepresentative value for the color components for each of the shortimage signals. Here, the representative value is intended to mean, forexample, a maximum value, or an average value, viz., a brightness valueof the complementary color image signal, or the like. Further, therepresentative value may be a brightness value of a primary color imagesignal, which can be computed in accordance with an expression 8described below.Brightness Value=0.3*R+0.6*G+0.1*B  (Expression 8)

The first level converting factor calculating unit 83 is operative tocalculate a level converting factor for each of the long image signalson the basis of the representative value calculated by the firstrepresentative value calculating unit 82 a. The second level convertingfactor calculating unit 84 is operative to calculate a level convertingfactor for each of the short image signals on the basis of therepresentative value calculated by the second representative valuecalculating unit 82 b.

The weighting factor calculating unit 85 is operative to input therepresentative value calculated by the first representative valuecalculating unit 82 a and calculate a weighting factor for each of thelong image signals on the basis of the representative value. The firstmultiplying unit 86 a is operative to multiply the level convertingfactor calculated by the first level converting factor calculating unit83 by the weighting factor calculated by the weighting factorcalculating unit 85, and the second multiplying unit 86 b is operativeto multiply the level converting factor calculated by the second levelconverting factor calculating unit 84 by the weighting factor calculatedby the weighting factor calculating unit 85. Likewise, the thirdmultiplying unit 87 a, the fourth multiplying unit 87 b, and the addingunit 88 operate in the same manner as the third multiplying unit 16 a,the fourth multiplying unit 16 b, and the adding unit 17 operate in thefirst embodiment.

As will be seen from the foregoing description, it will be understoodthat the present embodiment of the image processing apparatus accordingto the present invention ensures that the ratio of respective colorcomponents remains unchanged and thus can obtain high quality colorimages of a wide dynamic range with the hue of the color imagesmaintained constant as well as prevent circuits forming part thereoffrom being increased in size due to the increase in the number of bitsto make the image signals substantially close to one another and thedivision process to calculate the rate of change of the brightnesssignals as in the case of the conventional image processing apparatus,resulting from the fact that the image processing apparatus is operativeto multiply each of the image signals by the factors calculated based onthe representative value for the color components.

FIG. 8 is a block diagram showing an imaging apparatus applicable to theembodiment of the image processing apparatus according to the presentinvention.

The imaging apparatus to be applied to any one of the embodiments of theimage processing apparatus according to the present invention is shownin FIG. 8 as comprising an imaging device 91, an analog processing unit92 for carrying out an image processing on an analog image signalreceived from the imaging device 91, an AD converting unit 93 forconverting the analog image signal into a digital image signal, and animage signal processing unit 94 for carrying out an image processing onthe digital image signal. The image signal processing unit 94 includes atiming control section 95 for synchronizing the image signal, a memory96, an image signal synthesizing section 97 for synthesizing long imagesignals with short image signals, and an image processing section 98.

The imaging device 91 is constituted by a solid-state image sensingdevice such as for example a CCD (charge-coupled device), a COS(complementary metal oxide semiconductor), or the like, and operative toconvert light into an electrical signal, viz., an image signal. Theanalog processing unit 92 is operative to carry out an image processingsuch as, for example, correlated double sampling (CDS), automaticamplitude control (AGC), and the like on the analog image signal. The ADconverting unit 93 is operative to convert the analog image signal intoa digital image signal to be inputted to the image signal processingunit 94.

The timing control section 95 forming part of the image signalprocessing unit 94 is operative to temporarily store the image signal inthe memory 96 to synchronize the long image signals with the short imagesignals in such a manner that the long image signals and the short imagesignals forming part of the same image signals indicative of the objectare processed concurrently, the image signal synthesizing section 97 isoperative to synthesize the long image signals with the short imagesignals, and the image processing section 98 is operative to carry outan image signal processing on the image signal thus synthesized andoutput the image signals therethrough.

Further, in the imaging apparatus, the imaging device 91, the analogprocessing unit 92, and the AD converting unit 93 are operative at arate several times as fast as that of images in respective exposuretimes so as to obtain image signals indicative of the images differentfrom one another in exposure time, and the timing control section 95 isoperative to temporarily store the image signals in the memory 96 tosynchronize the image signals with one another so as to process theimage signals different from one another in exposure time.

This leads to the fact that memory capacity required to selectivelyswitch screens respectively forming part of the image signals differentfrom one another in exposure time is calculated in accordance with anexpression 9 as follows.Memory Capacity=Data for one whole Screen*(the number of image signalsdifferent from one another in exposure time−1)  (Expression 9)

Likewise, memory capacity required to selectively switch linesrespectively forming part of the image signals different from oneanother in exposure time is calculated in accordance with an expression10 as follows.Memory Capacity=Data for one Line*(the number of image signals differentfrom one another in exposure time−1)  (Expression 10)

Though it has been described in the above that the imaging apparatuscomprises one single imaging device 91, one single analog processingunit 92, and one single AD converting unit 93 electrically connected inseries as shown in FIG. 8, the imaging apparatus may comprise aplurality of imaging devices, analog processing units, and AD convertingunits to make a plurality of lines each having one imaging device, oneanalog processing units, one AD converting unit electrically connectedin series with one another. The imaging apparatus thus constructed caneliminate the need of the timing control section 95 and the memory 96 tosynchronize the image signals different in exposure time from oneanother as well as obtain a plurality of images different from oneanother in exposure time with no time deviation.

Further the imaging apparatus may further comprise a filter for limitingan amount of light, for example, a natural density filter, hereinlatersimply referred to as “ND filter” for limiting the signal levels of theimage signals without modifying their frequency characteristics. Thismeans that the imaging apparatus may comprise an ND filter in the formof checkered or striped pattern disposed in the vicinity of the imagingdevice 91 to obtain image signals different from one another in signallevel, which are substantially equivalent in effect to the image signalsdifferent from one another in exposure time. The imaging apparatus thusconstructed can decrease time deviation and facilitate thesynchronization process carried out at the timing control section 95resulting from the fact that the ND filter is operative to alternatelyoutput high-intensity image signal and low-intensity image signal.

As will be seen from the foregoing description, it will be understoodthat the imaging apparatus applied to any one of the embodiments of theimage processing apparatus can obtain the images of a wide dynamic rangeso as to prevent dark portions of the object displayed on, for example,the monitor from becoming monotonously black and ambiguous and brightportions of the object displayed on the monitor from becomingmonotonously white in a simple manner, resulting from the fact that theimaging apparatus is operative to convert a plurality of imagesdifferent in brightness from one another into a plurality of imagesignals, and synthesize the image signals.

While it has been described in the above about the image processingapparatus according to the present invention, the present invention maybe implemented by an image processing program executable by a machine toperform a set of process steps necessary to obtain high quality imagesof a wide dynamic range in a simple manner while reducing processingdelays and computing errors, and thus loss of minute image information.

Further, the aforementioned program may be embodied in a program storagemedium readable by a machine to perform a set of process steps necessaryto obtain high quality images of a wide dynamic range in a simple mannerwhile reducing processing delays and computing errors, and thus loss ofminute image information.

INDUSTRIAL APPLICABILITY OF THE PRESENT INVENTION

As will be seen from the foregoing description, it is to be understoodthat the image processing apparatus according to the present inventioncan reduce the number of computations carried out on image signals andstill obtain high quality images of a wide dynamic range so as toprevent dark portions of the object displayed on, for example, themonitor from becoming monotonously black and ambiguous and brightportions of the object displayed on the monitor from becomingmonotonously white in a simple manner without increasing the size ofcircuits forming part thereof as well as reduce accumulated processingdelays and computing errors, and thus decrease loss of minute imageinformation, resulting from the fact that the image processing apparatusis operative to input image signals different from one another inexposure time and multiply the level converting factors by the weightingfactors before the image signals are multiplied by the level convertingfactors.

1. An image processing apparatus, comprising: image signal inputtingmeans for inputting a plurality of image signals different from oneanother in exposure time; level converting factor calculating means forcalculating a level converting factor for each of said image signals;weighting factor calculating means for calculating a weighting factorfor each of said image signals; and adding means for adding up saidimage signals respectively multiplied by said level converting factorsmultiplied by said weighting factors.
 2. An image processing apparatusas set forth in claim 1, in which said level converting factorcalculating means is operative to calculate said level converting factorfor each of said image signals in such a manner that said image signalsrespectively multiplied by said level converting factors aresubstantially equal to one another in signal level under the conditionthat said signal levels of said image signals are less than apredetermined value, and said image signals respectively multiplied bysaid level converting factors are reduced in signal level under thecondition that said signal levels of said image signals are above saidpredetermined value.
 3. An image processing apparatus as set forth inclaim 1, which further comprises: high frequency component extractingmeans for allowing only high frequency components of said image signalsgreater than a predetermined frequency level to be passed therethrough;and high frequency level converting factor calculating means forcalculating a high frequency level converting factor for each of saidhigh frequency components of said image signals passed by said highfrequency component extracting means, and in which said adding means isoperative to multiply by said weighting factor the sum of each of saidimage signals multiplied by said level converting factor and said highfrequency component of each of said image signals multiplied by saidhigh frequency level factor.
 4. An image processing apparatus as setforth in claim 1, in which each of said image signals includes colorcomponents, and which further comprises: representative valuecalculating means for calculating a representative value for said colorcomponents for each of said image signals, and in which said levelconverting factor calculating means is operative to calculate a levelconverting factor for each of said image signals on the basis of saidrepresentative value, and said weighting factor calculating means isoperative to calculate a weighting factor for each of said image signalson the basis of said representative value.
 5. An image processingapparatus as set forth in claim 1, in which said image signal inputtingmeans is operative to input a plurality of image signals from an imagingdevice operative to temporarily store therein image signals ofrespective exposure times for respective time periods in such a mannerthat said image signals are synchronized with one another.
 6. A programexecutable by an image processing apparatus to carry out: an imagesignal inputting step of inputting a plurality of image signalsdifferent from one another in exposure time; a level factor calculatingstep of calculating a level converting factor for each of said imagesignals; a weighting factor calculating step of calculating a weightingfactor for each of said image signals; and an adding step of adding upsaid image signals respectively multiplied by said level convertingfactors multiplied by said weighting factors.
 7. A program executable byan image processing apparatus as set forth in claim 6, in which saidlevel factor calculating step is of calculating said level convertingfactor for each of said image signals in such a manner that said imagesignals respectively multiplied by said level converting factors aresubstantially equal to one another in signal level under the conditionthat said signal levels of said image signals are less than apredetermined value, and said image signals respectively multiplied bysaid level converting factors are reduced in signal level under thecondition that said signal levels of said image signals are above saidpredetermined value.
 8. A program executable by an image processingapparatus as set forth in claim 6, in which said level factorcalculating step is of calculating said level converting factor for eachof said image signals on the basis of predetermined data representativeof level converting factor characteristics.
 9. A storage medium havingstored therein a program executable by an image processing apparatus tocarry out: an image signal inputting step of inputting a plurality ofimage signals different from one another in exposure time; a levelfactor calculating step of calculating a level converting factor foreach of said image signals; a weighting factor calculating step ofcalculating a weighting factor for each of said image signals; and anadding step of adding up said image signals respectively multiplied bysaid level converting factors multiplied by said weighting factors. 10.A storage medium as set forth in claim 9, in which said level factorcalculating step is of calculating said level converting factor for eachof said image signals in such a manner that said image signalsrespectively multiplied by said level converting factors aresubstantially equal to one another in signal level under the conditionthat said signal levels of said image signals are less than apredetermined value, and said image signals respectively multiplied bysaid level converting factors are reduced in signal level under thecondition that said signal levels of said image signals are above saidpredetermined value.
 11. A storage medium as set forth in claim 9, inwhich said level factor calculating step is of calculating said levelconverting factor for each of said image signals on the basis ofpredetermined data representative of level converting factorcharacteristics.